Summary

Repairs TaskRunner Durable Object / D1 task lifecycle reconciliation so dead work no longer sits in_progress / awaiting_followup until the 8-hour hard timeout. Adds a separate configurable 24-hour absolute runaway-cost ceiling that fails even demonstrably live tasks, bounding unbounded compute without changing the liveness-gated semantics of TASK_RUN_HARD_TIMEOUT_MS.

Production evidence (2026-07-11, read-only D1/observability): Since 2026-06-01 there were 57 TaskRunner DO completed but task still in 'in_progress' warnings and 37 awaiting_followup 480-minute hard-timeout failures. All 34 D1 rows with the exact 480-minute message were conversation-mode tasks whose workspaces are now deleted; 32 still had agent_sessions.status='running', proving that status is stale liveness. Soft-timeout recovery kept skipping them on shared-node heartbeat, which proves host — not task — liveness. Dominant exit path is normal post-handoff liveness drift (no bounded-window aborted_by_recovery records found), but the recovery race is code-grounded and closed here too.

Changes:

• Composite task-scoped liveness probe (getTaskRuntimeLiveness, stuck-tasks.ts): requires live workspace (running/recovery) and healthy node with recent heartbeat and a task-scoped ACP session with a recent heartbeat. A shared-node heartbeat is never sufficient. Returns { live, conclusive, reason }; inconclusive/errored liveness never fails a task (fail-safe protects genuinely-running long work).
• Prompt reconciliation of DO-completed + D1-active mismatches: transitions to a clear terminal state with sanitized reason only when composite liveness is conclusively dead, gated by a configurable TASK_DO_MISMATCH_GRACE_MS (5 min) instead of waiting for the 8-hour timeout. awaiting_followup is now covered alongside in_progress.
• aborted_by_recovery race closure (state-machine.ts): the zero-row optimistic delegated→in_progress update now re-reads authoritative D1 status — in_progress → DO marks running/done; already-terminal/missing → DO completes without clobbering a legitimate terminal callback; still-transient (superseded) → failTask with a clear message.
• Idempotent terminal callback (callback.ts): a repeated same-terminal callback reuses cleanupTerminalTaskResourcesOrThrow and returns the task instead of throwing a transition conflict.
• Configurable, bounded control loop: new shared DEFAULT_* constants + env fallbacks; candidate query is LIMIT-bounded and the ACP read is bounded per candidate. TASK_RUN_ABSOLUTE_CEILING_MS defaults to 24 hours and terminates even a live-runtime task before any ACP liveness probe, providing a pure runaway-cost backstop.
• Removed now-dead getTaskNodeId / isNodeHeartbeatRecent helpers; corrected diagnostic elapsed-time to use started_at for in_progress/awaiting_followup.
• Docs: configuration.md, .env.example, and the canonical env-reference source document the new configuration, including TASK_RUN_ABSOLUTE_CEILING_MS=86400000.

Validation

• Option 2 focused node-pool regression: 20 passed; mutation check proved the 25-hour live-task test fails when the ceiling branch is removed.
• pnpm typecheck (api clean after shared/providers/cloud-init build)
• pnpm lint on changed files — 0 errors (2 pre-existing non-null-assertion warnings outside this diff)
• Targeted tests: stuck-tasks, task-runner-state-machine, recovery-resilience, task-callback-recoverable-error — 90 passed (node pool + separate callback pool)
• Miniflare workers-pool test tests/workers/scheduled-stuck-tasks.test.ts — passed in the rebased local full pnpm test run and in CI Test.
• Control-loop candidate-volume note (rule 47): This PR changes the sweep candidate-selection behavior for active D1 task rows and adds a per-candidate ACP liveness read for liveness-gated branches. Expected candidate volume: bounded by STUCK_TASK_MAX_CANDIDATES_PER_SWEEP (default 100) via LIMIT. Worst-case per-candidate cost: one indexed workspace+node D1 read; the more expensive ACP-session read (listAcpSessions, bounded by TASK_LIVENESS_MAX_ACP_SESSIONS=5) only runs after cheap workspace/node liveness checks pass, so dead workspaces short-circuit before it. DO getStatus() mismatch probe is gated by TASK_DO_MISMATCH_GRACE_MS. Every selected candidate now has a terminal escape path: without the absolute ceiling, a live-forever task with stale updated_at would sort to the front of the LIMIT window on every tick, burn a liveness probe, and crowd out genuinely stuck work. The ceiling terminates that rule-47 immortal candidate before the DO probe.

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — pending (see coordination note below)
• Live app verified via Playwright
• Existing workflows confirmed working
• New feature/fix verified on staging
• Infrastructure verification — N/A: no infra changes (no cloud-init / VM agent / DNS / TLS / scripts/deploy changes)
• Mobile/desktop — N/A: no UI changes

Staging Verification Evidence

Staging verification is being completed by follow-up task 01KXAX6D6F5AFKJQDC6WT7DGZK after rebase onto current main and force-push. The needs-human-review label remains until staging evidence is recorded and all gates are green.

End-to-End Verification

Data Flow Trace

1. Cron scheduled() → recoverStuckTasks(env) (apps/api/src/scheduled/stuck-tasks.ts) selects bounded active/awaiting_followup candidates.
2. Per candidate → getTaskRuntimeLiveness(env, task) reads workspaces+nodes (D1) then task-scoped projectDataService.listAcpSessions (ProjectData DO).
3. DO-completed mismatch branch → env.TASK_RUNNER.get(...).getStatus(); if completed && D1 non-terminal && liveness conclusive-dead → mark stuck.
4. Stuck → failTask path writes terminal D1 status + taskStatusEvents + cleanupTerminalTaskResourcesOrThrow.
5. Independent race path: state-machine.ts:transitionToInProgress zero-row update re-reads tasks.status and converges DO/D1.

Untested Gaps

Full cron→DO→D1 reconciliation is covered by the Miniflare workers-pool test scheduled-stuck-tasks.test.ts (recovery interleaving, callback idempotency, dead-mismatch, live-negative) — runs in CI. Live-agent negative case and aborted_by_recovery interleave have node-pool unit coverage that passed locally.

Post-Mortem

What broke

Dead conversation-mode tasks (workspace deleted, agent gone) remained in_progress/awaiting_followup for up to 8 hours because the reconciler only logged DO-completed/D1-active mismatches and its soft-timeout grace trusted shared-node heartbeat as task liveness.

Root cause

stuck-tasks.ts used node heartbeat (host liveness) as a proxy for task liveness and treated the DO/D1 mismatch as informational; the state-machine.ts zero-row optimistic update completed the DO without re-reading D1.

Class of bug

Reporter/liveness-scope mismatch — a coarse-grained signal (shared-node heartbeat, DO "done") used as authority for a finer-grained entity (per-task/workspace/agent liveness), with a check-then-act race across the DO/D1 boundary.

Why it wasn't caught

No test asserted the DO-completed + D1-non-terminal reconciliation against composite liveness; existing coverage stopped at node heartbeat.

Process fix included in this PR

Adds cross-runtime regression coverage (workers-pool) for DO-completed+D1-active, recovery interleaving, live-negative, and callback idempotency, exercising the exact liveness contract that was violated. (No new .claude/rules file — the existing rules 02/11/47 already cover this class; tests are the concrete guard.)

Post-mortem file

Task record tasks/active/2026-07-11-taskrunner-d1-lifecycle-reconciliation.md; idea 01KT90PKF6167SXZ9YZY0R26MM.

Specialist Review Evidence (Required for agent-authored PRs)

• All local reviewers completed; every concrete finding fixed and pushed
• The prior HIGH design decision is resolved: Option 2 implemented; needs-human-review remains for Raphael handoff

RESOLVED: Option 2 implemented

A separate TASK_RUN_ABSOLUTE_CEILING_MS backstop now defaults to 24 hours and fails a task even when its ACP session keeps a fresh heartbeat. This intentionally deviates from the task literal "prevent early failure regardless of elapsed duration" wording.

Justification:

1. It caps unbounded compute for a looping agent that keeps its heartbeat fresh. No other SAM mechanism does: reconciliation is idle-gated and never fires on a chattering loop, while node max-lifetime cleanup skips nodes with active workspaces.
2. Rule 47 requires every sweep candidate to have an exit path. Without the ceiling, a live-forever task becomes an immortal stuck-task-sweep candidate: stale updated_at sorts it to the front of the LIMIT window every tick, burning a probe and crowding out genuinely stuck tasks. The ceiling check runs before the liveness probe.
3. SAM already has precedent for failing provably live tasks when the compaction-loop detector identifies runaway token spend.

TASK_RUN_HARD_TIMEOUT_MS semantics are unchanged; the new, higher ceiling is independent. awaiting_followup is not exempt.

Exceptions (If any)

• Scope: none currently open for specialist review evidence.
• Rationale: workers-pool coverage now passed in local full pnpm test and CI.
• Expiration: N/A

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• cross-component-change
• business-logic-change
• docs-sync-change

External References

N/A: internal reconciliation logic. Grounded in read-only production D1/observability evidence (documented in the task file + idea 01KT90PKF6167SXZ9YZY0R26MM) and existing .claude/rules/47-control-loop-io-budget.md, .claude/rules/11-fail-fast-patterns.md.

Codebase Impact Analysis

• apps/api/src/scheduled/stuck-tasks.ts (reconciler + liveness probe)
• apps/api/src/durable-objects/task-runner/state-machine.ts (race closure)
• apps/api/src/routes/tasks/callback.ts (idempotent terminal callback)
• apps/api/src/env.ts, packages/shared/src/constants/task-execution.ts + index.ts (config)
• apps/www/.../reference/configuration.md, apps/api/.env.example (docs)

Documentation & Specs

Updated apps/www/src/content/docs/docs/reference/configuration.md and apps/api/.env.example for the three new env vars.

Constitution & Risk Check

• Principle XI (no hardcoded values): all new thresholds/limits are DEFAULT_* constants with env overrides.
• Rule 47 (control-loop I/O budget): candidate LIMIT bound, tiered/gated per-candidate reads, terminal/skip escape path for every candidate.
• Rule 11 (fail-fast identity/liveness at boundaries): liveness is fail-safe — inconclusive never fails a task.
• Key risk: false-positive early failure of a live long task — mitigated by requiring conclusive-dead composite liveness and the conclusive fail-safe.